Evaluating Evaluation Measures

Model selection measures such as hold-out set error, cross-validation error, leave-one-out error etc. are used to evaluate the performance of a classification algorithm on a given data set. To get an accurate estimate of the performance it is important that we choose the appropriate model selection measure or evaluation measure for the setting of interest. In this paper, we describe in brief a recently introduced methodology, which can be used to accurately and efficiently study the behavior of such evaluation measures in relevant settings. We also discuss the implications of having such a methodology as an exploratory tool and the potential challenges for the future.